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ABSTRACT 


The  Center  for  Science,  Technology,  Engineering,  and  Mathematics  Education  (CSTEME)  was  one  of  four 
sponsored  centers  through  the  National  Institute  of  Science,  Space,  and  Security  Centers  located  on  the 
University  of  Colorado,  Colorado  Springs  (UCCS)  campus  and  initially  funded  by  the  U.S.  Air  Force  Office  of 
Scientific  Research  (AFOSR).  Through  its  Partnership  in  Innovative  Preparation  for  Educators  and  Students 
(PIPES)  program,  CSTEME  responded  to  the  lagging  performance  and  retention  of  students  in  science  and 
math  through  innovative  and  supportive  partnerships  with  parents,  educators,  and  professionals. 
Leveraging  on-campus  faculty  as  well  as  the  technology  and  military  industries  that  are  so  prevalent  in  the 
Pikes  Peak  area,  CSTEME  aspired  to  attract  and  encourage  a  new  generation  of  creative,  artistic,  and 
innovative  students  to  solve  our  future  problems  related  to  science,  technology,  engineering,  and 
mathematics  (STEM).  Underlying  all  PIPES  activity  was  a  solid  evaluation  framework  that  measured 
student  interest  and  retention  in  STEM  subjects  through  longitudinal  tracking  of  students  from  6th  through 
12th  grade  and  assessing  teachers  who  completed  PIPES  professional  development  programs  over  4 
years.  PIPES  researchers  collected  and  analyzed  both  qualitative  and  quantitative  data  from  students, 
teachers,  and  parents  related  to  PIPES  program  effectiveness  in  stimulating  interest  in  STEM  subjects  and 
long-term  attraction  and  retention  in  STEM  careers. 
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INTRODUCTION 


Through  the  Partnership  in  Innovative  Preparation  for  Educators  and  Students  (PIPES)  program,  the 
Center  for  STEM  Education  at  the  University  of  Colorado  Colorado  Springs  responded  to  the  lagging 
performance  and  retention  of  middle  and  high  school  students  in  science  and  math  through  the  use  of 
innovative  and  supportive  partnerships  with  parents,  educators,  and  professionals.  Leveraging  on-campus 
faculty  as  well  as  the  technology  and  military  industries  that  are  so  prevalent  in  the  Pikes  Peak  area,  the 
PIPES  program  aspired  to  attract  and  encourage  a  new  generation  of  creative,  artistic,  and  innovative 
students  to  solve  our  future  problems  related  to  science,  technology,  engineering,  and  mathematics 
(STEM).  Funding  received  from  AFOSR  in  2007  was  used  to  initiate  the  PIPES  effort.  The  current  award 
(FA9550-09-1-0713),  which  concluded  in  September  2013,  had  as  its  goal  the  "further  support  and 
extension  of  the  PIPES  program  efforts". 

PIPES  offered  an  array  of  programs  for  students  from  regional  middle  and  high  schools  that  provided 
participants  opportunities  to  explore  careers  in  STEM  fields.  The  program  had  impacted  1,480  students, 
63  science  teachers,  and  103  math  teachers,  most  of  whom  were  from  underserved  school  districts. 
These  students  and  teachers  were  introduced  to  innovative  courses  and  hands-on  workshops  that 
challenged  their  minds  in  a  creative  and  energetic  atmosphere.  When  the  combined  impact  of  teacher  and 
student  programs  is  explored  the  PIPES  program  both  directly  and  indirectly  reached  over  23,000  students 
in  the  Pikes  Peak  region  and  throughout  Colorado. 

Underlying  all  PIPES  activity  was  a  solid  evaluation  framework  that  measured  student  interest  i  n 
STEM  subjects  and  persistence  in  STEM  study  through  longitudinal  tracking  of  students  from  6th  through 
12th  grade.  Additionally,  teachers  who  completed  PIPES  professional  development  programs  were 
surveyed  over  4  years.  PIPES  researchers  collected  and  analyzed  both  qualitative  and  quantitative  data 
from  students,  teachers,  and  parents  related  to  PIPES  program  effectiveness  in  stimulating  interest  in 
STEM  subjects  and  long-term  attraction  and  retention  in  STEM  careers. 

This  evaluation  framework  measured  changes  in  student  attitude  toward  STEM,  their  STEM  career  interest, 
and  their  intention  to  enter  a  STEM  field.  Additionally,  STEM  retention,  and  academic  achievement  in 
STEM  subjects  were  evaluated.  The  results  of  this  research  provided  crucial  guidance  in  developing 
STEM  education  programs  that  increased  student  engagement,  performance,  and  retention  in  STEM 
subjects  from  6th  through  12th  grades.  Longitudinal  research  on  this  scale  is  rare,  but  necessary  to 
adequately  address  the  STEM  pipeline  shortage  facing  our  nation.  In  the  future,  these  outcomes  will 
inform  other  ongoing  STEM  education  research  projects  and  will  provide  guidance  to  educational  and 
policy  leaders. 

PIPES  research  provided  tangible  results  to  the  Air  Force  by  directly  addressing  the  concern  about 
maintaining  a  workforce  with  the  requisite  STEM  skills  needed  to  support  the  Air  Force's  mission. 
Furthermore,  PIPES  addressed  the  STEM  professional  shortage  by  researching  the  most  effective  ways 
of  developing  STEM-cognizant  students  who  pursue  a  STEM  degree  after  graduation. 


STUDENT  MODEL 


The  PIPES  student  research  model  (see  Figure  1)  was  based  on  a  rigorous  review  of  the  STEM  educational 
research  literature.  The  model  emerged  from  best  practice  recommendations  from  nationally 
recognized  STEM  education  organizations  such  as  the  National  Science  Foundation,  the  National  Science 
Teachers  Association,  Project  2061,  the  American  Association  for  the  Advancement  of  Science  (AAAS), 
the  National  Academies,  and  the  National  Research  Council.  Based  on  the  prior  work  of  these 
organizations  the  PIPES  program  model  was  formulated  which  encompassed  inquiry  learning,  longitudinal 
student  engagement,  and  industry  engagement.  PIPES  student  programs  were  built  around  the  5E 
model  of  inquiry  (Flick,  1995)  shown  to  engage  students  at  deep  levels  of  questioning,  and  participation 
(Banilower,  Cohen,  Pasley,  &  Weiss,  2008).  In  addition,  PIPES  student  programs  utilized  problem-based 
(Lampert,  2001),  authentic  learning  (Albanese  &  Mitchell,  1993)  constructivist  pedagogies  (Lombardi  & 
Oblinger,  2007)  requiring  collaboration  with  peers  within  a  distinct  learning  context  that  students  can 
easily  relate  to  (Stanley  &  Waterman,  2005).  The  longitudinal  nature  of  student  and  parent  support  was 
found  to  substantially  increase  student  retention  and  interest  in  STEM  content  (George,  2006). 

Figure  1  -  PIPES  Student  Research  Model 
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PIPES  Evaluation  Questions: 

This  project  was  designed  to  address  the  following  evaluation  questions: 

•  To  what  degree  did  PIPES  student  programs  improve  student  confidence  and  motivation  to  pursue 
advanced  STEM  courses  in  high  school? 

•  To  what  degree  did  PIPES  student  programs  improve  student  confidence  and  motivation  to  pursue 
a  STEM  major  in  college? 

•  To  what  degree  did  PIPES  student  programs  improve  student  confidence  and  motivation  to  pursue 
a  STEM  career? 

•  To  what  degree  did  PIPES  programs  increase  the  number  of  students  declaring  a  STEM  major  after 
high  school  graduation? 


PIPES  Evaluation  Model  Variables: 

This  study  utilized  two  constructs  as  predictor  variables.  These  constructs  included: 

•  Student  and  Family  Characteristics  -  demographics  (e.g.,  age/grade,  gender,  race,  ethnicity, 
household  income),  parents'  education  /  occupation,  and  parents'  expectations  for  college  and; 

•  STEM  Learning  Variables  -  PIPES  workshops  and  summer  camps  intensity,  measured  in  hours  of 
participation  in  PIPES,  duration,  and  breadth-variety  of  programs  (STEM-dose). 

The  constructs  used  as  the  outcome  variables  included: 

•  Student  Variables  -  motivation  for  STEM  career,  science  &  math  self-concept  and  self-efficacy, 
attitudes  toward  science  and  career  in  science,  science  confidence  &  knowledge,  social  niche, 
science  and  math  achievement,  parent  attitudes  toward  STEM,  family  encouragement,  peer 
influence,  and  teacher  encouragement. 

•  Student  Major  Choice  -  whether  or  not  a  PIPES  student  selected  a  STEM  major  after  graduation 
from  PIPES. 

Details  on  the  specific  constructs  and  survey  instruments  used  can  be  found  in  Appendix  A. 

PIPES  Programs: 

The  PIPES  model  provided  a  developmental^  appropriate  longitudinal  sequence  of  hands-on  STEM 
workshops  for  academically  motivated  students  from  predominantly  underserved  school  districts.  PIPES 
students  engaged  in  a  maximum  of  424  hours  of  peer  collaboration,  lab  and  field  work,  engineering  design, 
and  innovative  problem  solving  alongside  graduate  level  facilitators,  master  teachers,  university  faculty, 
and  industry  professionals.  Starting  in  6th  grade,  PIPES  students  were  exposed  to  a  wide  array  of  STEM 
topics  (see  Appendix  2).  As  they  transitioned  to  high  school,  students  were  engaged  in  deeper  learning 
through  multiple,  in-depth  experiences  in  a  STEM  related  topic  of  their  choice.  PIPES  programs  included: 

•  Mind  Quest:  A  series  of  middle  and  high  school  workshops  held  four  times  per  academic  year 
covering  a  variety  of  STEM  topics  (e.g.,  computer  science,  engineering,  food  science,  and 
biotechnology). 

•  STEM  in  Real  Life:  A  two-day  summer  camp  for  middle  school  students  designed  to  allow  each 
participant  to  experience  the  wide  breadth  of  STEM  subjects  present  in  society  today. 

•  FLITE  (First  in  Leadership,  Innovation,  Technology,  and  Engineering):  A  week-long  summer  camp 
for  high  school  students  to  engage  in  research,  engineering,  and  innovation  by  working  with 
industry  mentors  and  university  graduate  students. 

•  Jumpstart  STEM:  A  week-long  summer  camp  for  high  school  students  which  offers  students  the 
opportunity  to  apply  their  STEM  knowledge  to  real-world  scenarios.  Each  scenario  is  built 
around  a  central  story  in  which  students  answered  a  critical  question  through  experimentation  and 
data  analysis. 

•  Bridge  to  Engineering:  A  week-long  summer  camp  for  high  school  students  interested  in  pursuing 
an  engineering  major  after  graduation.  Students  work  in  collaborative  teams  to  address  an 
engineering  design  challenge.  Student  test  their  designs  in  the  field  then  present  their  findings  to  a 
panel  of  industry  experts.  The  design  challenges  encompassed  all  major  engineering  disciplines. 

•  Math  Bridge:  The  capstone  experience  for  PIPES  high  school  students  which  offered  students  the 
opportunity  to  assess  their  math  skills  and  remediate  any  deficiencies. 
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DESCRIPTION  OF  PIPES  STUDENTS 


Student  Numbers  and  Demographics: 

The  number  of  participating  students  started  at  53  in  2008  and  grew  to  1,480  PIPES-enrolled  students  in 
2012-13  (see  Figure  2).  More  than  half  of  the  students  came  from  underserved,  low  socioeconomic  status 
school  districts  serving  the  Colorado  Springs  region.  A  significant  number  of  PIPES  students  (46%)  were 
students  who  will  be  the  first  in  their  families  to  attend  college  (known  as  first-generation  students). 

Figure  2  -  PIPES  Student  Yearly  Enrollment  Growth 
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PIPES  Student  Demographics  (n  =  1,480) 
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Figure  5 
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Figure  7 


RESULTS 


PIPES  Evaluation  Question  Results: 

Through  the  generous  support  of  AFOSR,  the  PIPES  project  produced  significant  results  as  analyses 
showed  increased  interest  and  confidence  to  pursue  advanced  STEM  classes  in  high  school,  STEM  courses 
in  college,  a  STEM  major,  and  a  STEM  career. 

These  data  were  produced  from  two  types  of  student  surveys.  The  first  type  was  a  longitudinal  survey 
given  before  PIPES  involvement  (pre-survey)  and  after  each  year  of  PIPES  involvement  (post-survey).  The 
second  type  was  a  retrospective  pre/post  survey  given  after  each  camp  and  workshop.  In  this  survey, 
students  recorded  their  thoughts,  feelings,  and  attitudes  toward  STEM  before  the  camp  then  after  the 
camp.  Paired  samples  t-  tests  were  conducted  to  evaluate  the  impact  of  the  camp  or  workshop  on 
student  perception,  motivation  and  confidence  linked  to  the  desire  to  pursue  STEM  further  in  high  school 
and  college. 

Qualitatively,  comments  from  students,  parents,  and  teachers  showed  a  high  level  of  enthusiasm  for 
PIPES  programs  (see  participant  comments,  Appendix  3).  Post-program  survey  results  revealed  that  PIPES 
programs  had  a  positive  influence  on  students'  internal  motivation  to  study  STEM,  student  confidence  in 
their  ability  to  do  STEM  courses,  their  general  STEM  knowledge,  and  achievement  (see  Figures  8-11). 
These  finding  indicated  that  the  PIPES  model  of  student  outreach  is  an  effective  program  in  increaseing 
students'  interest  in  STEM  courses  and  careers. 


Question  1:  To  what  degree  did  PIPES  programs  increase  the  number  of  students  declaring  a 
STEM  major  after  high  school  graduation  ? 

Longitudinally,  6%  of  the  PIPES  students  (58%  male,  42%  female)  have  graduated  high  school. 
Over  83%  of  these  students  went  on  to  study  in  college  (94%  of  which  in  Colorado),  higher  than 
the  national  percentage  (68%  in  2011;  Bureau  of  Labor  Statistics,  2012)  and  in  Colorado  (63%  in 
2008;  The  National  Center  for  Higher  Education  Management  Systems,  2013).  Additionally,  59% 
of  the  students  who  went  to  college  attended  the  University  of  Colorado  Colorado  Springs 
(UCCS),  with  an  additional  19%  attending  Pikes  Peak  Community  College  (a  UCCS  pipeline). 
Further,  45%  of  the  students  who  entered  college  declared  a  STEM  major,  triple  the  national 
rate  (15%  in  2006;  National  Center  for  Education  Statistics,  2009). 


Figure  8 

DECLARED  A  STEM  MAJOR 


*National  Center  for  Education  Statistics,  2009 
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Question  2:  To  what  degree  did  PIPES  student  programs  improve  student  confidence  and 
motivation  to  pursue  advanced  STEM  courses  in  high  school? 

PIPES  camps  and  workshops  significantly  improved  student  motivation  to  pursue  advanced 
STEM  courses  in  high  school.  Additionally,  PIPES  programs  improved  student  confidence  that 
they  could  succeed  in  these  advanced  courses  (see  Figure  9).  Both  of  these  measures  are 
predictors  of  future  STEM  retention  and  success  after  high  school  graduation. 


Figure  9  (n  =  69,  p  <  .01) 


Question  3:  To  what  degree  did  PIPES  student  programs  improve  student  confidence  and 
motivation  to  pursue  a  STEM  major  in  college? 

PIPES  camps  and  workshops  significantly  improved  student  motivation  and  confidence  to  pursue 
a  STEM  major  in  college.  Additionally,  the  number  of  students  planning  on  majoring  in  a  STEM 
field  significantly  increased  pre  to  post  (see  Figure  10). 


Figure  10  (n  =  69,  Motivation  significant  at  p  <  .01  level) 
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Question  4:  To  what  degree  did  PIPES  student  programs  improve  student  confidence  and 
motivation  to  pursue  a  STEM  career? 

PIPES  camps  and  workshops  significantly  improved  student  motivation  to  pursue  a  STEM  career. 
However,  no  significant  change  was  observed  in  student  confidence  that  they  could  succeed  in  a 
STEM  career.  This  finding  is  relatively  common  in  previous  studies  and  is  partially  due  to  the 
student  coming  to  a  deeper  understanding  of  the  challenges  of  a  STEM  career  (see  Figure  11). 


Figure  11  (n  =70,  p  <  .01) 


Per  program,  PIPES  was  successful  in  increasing  student  motivation,  confidence,  and  knowledge  in 
STEM.  Overall,  each  program  and  camp  in  the  PIPES  menu  of  offerings  produced  an  increase  in  these 
three  measures  from  6th  -  12th  grades  (see  Figure  12). 


Figure  12  -  %  Reporting  Increases  in  Science  Motivation,  Confidence,  &  Knowledge 
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EDUCATOR  MODEL 


The  second  tier  of  the  PIPES  program  emphasized  science  and  math  teacher  professional 
development.  Long-term  STEM  retention  is  heavily  dependent  on  qualified  and  enthusiastic  teachers 
utilizing  high  engagement  strategies  to  retain  student  interest  in  STEM  subjects  (National  Research 
Council,  2005).  The  basis  of  the  PIPES  teacher  professional  development  model  was  the  development  of 
professional  learning  communities  (PLC's)  in  which  teachers  could  learn  from  the  instructors  and  their 
peers  through  ongoing  professional  development  which  included: 

PIPES  Science  Educator  Academy:  This  professional  development  opportunity  provided 
secondary  science  teachers  with  high  quality,  inquiry-based  science  content  and  instructional 
strategies.  These  courses  were  taught  by  UCCS  faculty  and  veteran  master  science  teachers. 

Pikes  Peak  Math  Teachers'  Circle:  This  professional  development  academy  provided  secondary 
math  teachers  with  the  opportunity  to  engage  together  as  a  cohort  for  one  academic  year  in 
high  level  mathematical  reasoning,  problem  solving,  and  applied  skill  development. 

PIPES  Teacher  Program  Evaluation  Questions: 

Question  1:  To  what  degree  did  participation  in  the  PIPES  Science  Educator  Academy  (PSEA) 
improve  teachers'  confidence  in  using  inquiry,  STEM-based  practices  in  their  science 
classrooms? 

Question  2:  To  what  degree  did  participation  in  the  PIPES  Pikes  Peak  Math  Teacher's  Circle 
(PPMTC)  improve  teachers'  confidence  and  ability  to  use  inquiry,  problem-based  practices  in 
their  math  classrooms? 

PIPES  Evaluation  Model  Variables: 

The  predictor  variable  used  in  this  evaluation  was  whether  a  teacher  had  completed  the  PIPES 
professional  development  program.  Additional  demographic  data  were  collected  on  the  teacher  and 
school  for  future  analysis. 

The  constructs  used  as  the  outcome  variables  included: 

Teacher  self-efficacy  -  the  degree  to  which  the  teacher  feels  they  are  able  to  accomplish 
specific  inquiry  learning  tasks  with  their  students. 

Teacher  confidence  -  the  degree  to  which  the  teacher  feels  they  are  able  to  use  inquiry  learning 
strategies  in  their  classrooms  resulting  in  a  positive  outcome. 

Teacher  instructional  practice  -  the  degree  to  which  the  teacher  applies  the  inquiry  strategies  in 
their  classroom. 
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Description  of  PIPES  Teachers 


Teacher  Numbers: 


The  number  of  participating  teachers  began  in  2008  with  15  science  teachers,  each  having  an  impact  on 
an  average  of  150  students  per  year,  which  exceeded  2,000  students  annually.  In  2009,  the  Pikes  Peak 
Math  Teacher's  Circle  was  added  with  a  cohort  size  of  38  (see  Figure  13-  next  page).  By  2012,  PIPES 
programs  had  instructed  82  math  teachers  and  63  science  teachers,  for  a  total  student  impact  in 
excess  of  20,000  students  annually.  32%  percent  of  the  teacher  who  participated  taught  in  schools  in 
which  the  majority  of  students  qualified  for  free  and  reduced  lunch  services.  55%  of  the  teachers  who 
participated  taught  in  schools  with  minority  student  populations  of  at  least  30%. 

Figure  13  Teacher  Participation 
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PIPES  Teacher  Results: 

Since  the  first  PIPES  professional  development  programs  for  STEM  educators  in  2008,  early  data  indicated 
that  the  PIPES  model  improved  science  and  math  teachers'  confidence  in  using  inquiry-based  methods. 

Among  science  teachers  served,  statistically  significant  increases  were  observed  in  the  science  teachers' 
confidence  to  teach  science  content  using  inquiry-based  methods  after  being  in  the  program  for  one  year. 
Additionally,  science  teacher  reported  an  increase  in  confidence  that  implementing  their  newly  developed 
teaching  skills  would  increase  student  achievement  outcomes  (see  Figure  14  -  next  page). 


Figure  14 
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Among  math  teachers  served,  statistically  significant  increases  were  observed  in  their  confidence  to 
provide  guidance  to  their  students  and  to  develop  and  lead  an  investigative  culture  in  their  classrooms  (see 
Figure  15).  This  suggests  that  if  continued,  the  PIPES  model  of  teacher  professional  development  may 
result  in  hundreds  of  science  and  math  teachers  potentially  influencing  thousands  of  students  to  pursue  a 
STEM  major  and  career  field. 


Figure  15  -  Means  of  PIPES  Math  Teachers  Increased  Confidence  Measures 


^Denotes  Statistically  Significant  Differences  (n  =  80) 
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Overall,  the  PIPES  teacher  development  programs  significantly  improved  the  confidence  and  ability  of 
teachers  to  use  inquiry-oriented  approaches  in  their  classrooms.  These  approaches  are  important  to 
teaching  STEM  in  a  meaningful  and  applied  way  that  has  been  shown  to  increase  student  engagement  in 
STEM  subjects  and  retain  student  interest  in  STEM  majors  and  careers. 

Interestingly,  the  work  that  began  with  the  AFOSR  funding  is  being  sustained  through  the  use  of  PIPES 
teachers  becoming  STEM  education  experts  by  serving  as  instructors  for  PIPES  student  workshops. 
Additionally,  many  PIPES  teachers  have  moved  into  STEM  leadership  roles  in  their  schools  and  districts.  By 
PIPES  teachers  becoming  STEM  leaders  in  the  community,  the  initial  investment  by  AFOSR  has  produced 
tangible  dividends  toward  sustaining  the  use  of  STEM  methods  in  classrooms  across  the  region. 
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FUTURE  DIRECTION  &  SUSTAINABILITY 


PIPES  Dissemination  and  Replication: 

The  PIPES  research  team  has  made  concerted  efforts  to  disseminate  the  findings  produced  by  the  PIPES 
project  through  publications  and  presentations  around  the  nation  (see  Appendix  4).  Additionally,  replication 
of  PIPES  programs  has  always  been  an  important  goal  in  the  program  model.  Through  the  successes 
experienced  at  UCCS,  PIPES  has  been  able  to  work  with  many  other  STEM  organizations  across  the  state  to 
support  the  establishment  of  STEM  centers  based  on  the  PIPES  model.  Many  new  opportunities  for 
collaboration,  partnership,  and  funding  have  emerged  because  of  the  generous  support  by  AFOSR  to  establish 
the  PIPES  model.  The  work  that  began  in  Colorado  Springs  is  paying  dividends  nationally.  Projecting 
from  current  PIPES  student  and  educator  enrollment  patterns  and  anticipated  replication  opportunities, 
in  4  years  over  100,000  students  will  have  been  impacted  by  PIPES  programs,  with  the  potential  to 
reach  over  500,000  students  in  just  10  years.  Current  examples  include: 

•  For  the  past  three  summers,  the  STEM  In  Real  Life  middle  school  camp  was  replicated  at 
Otero  Junior  College,  a  small  community  college  located  in  the  underserved  southeastern 
corner  of  Colorado.  Sixty-three  students  were  served  and  preliminary  research  results 
showed  very  similar  patterns  to  the  UCCS  program,  suggesting  that  the  program  components 
are  portable  to  other  locations  around  the  country. 

•  STEM  programs  in  Pennsylvania,  Connecticut,  and  Colorado  have  used  the  PIPES  program  as  a 
model  for  their  own  STEM  program  structure.  In  many  cases,  these  centers  have  used  PIPES 
curriculum  developed  with  AFOSR  funding  for  their  own  camps. 

•  PIPES  STEM  curriculum  are  used  in  a  large  number  of  classrooms  across  the  Colorado  Springs 
region.  Schools  are  beginning  to  adopt  PIPES  curriculum  to  support  STEM  electives  and  classes 
embedded  in  the  school  day. 

•  Many  non-profit  outreach  programs  are  using  PIPES  curriculum  to  support  their  own  STEM 
outreach. 

•  The  quality  of  the  PIPES  program  has  generated  research  partnerships  with  the  University  of 
Wyoming,  Cornell  University,  and  California  Institute  of  Technology  (CalTech). 

PIPES  Future  Initiatives: 

Due  to  the  generous  support  of  AFOSR,  the  Center  for  STEM  Education  (CSTEME)at  UCCS  has  become  a 
significant  contributor  to  the  STEM  conversation  around  the  state.  Based  on  the  initial  funding  for 
programs,  research,  and  training  through  AFOSR,  CSTEME  is  now  involved  in  several  STEM  program 
evaluations,  curriculum  development  projects,  and  STEM  leadership  organizations.  Additionally,  the  initial 
success  of  the  PIPES  program  has  enabled  CSTEME  to  generate  over  $300,000  in  program  and  research 
funding.  Partnerships  include  the  Harris  Foundation,  ExxonMobil,  the  Anschutz  Foundation,  the  Women's 
Foundation,  the  Colorado  Governor's  Office  of  Information  Technology,  and  others.  The  PIPES  program, 
through  CSTEME  continues  to  grow  and  thrive  thanks  to  the  generous  funding  of  AFOSR  in  starting  the 
center  at  UCCS. 

Changes  for  Future  Success: 

A  project  of  the  scope  and  size  of  the  PIPES  project  creates  many  challenges  and  opportunities  to  learn 
from  mistakes.  Though  the  project  has  been,  and  continues  to  be  a  success,  there  have  been  many 
lessons  learned  from  the  process  of  establishing  a  regional  and  state  hub  for  STEM  learning.  First,  the 
importance  and  difficulty  of  managing  a  longitudinal  study  is  significant.  The  longitudinal  PIPES  dataset 
continues  to  be  a  work  in  progress  as  the  research  team  learns  how  to  best  encourage  PIPES  students  to 
complete  yearly  follow-up  surveys.  Student  retention  in  the  PIPES  program  is  very  high  at  63%  of  students 
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returning  for  more  than  two  programs.  The  difficulty  arises  in  engaging  families  to  fill  out  the  follow-up 
surveys  at  home.  Additionally,  preventing  survey  fatigue  among  students  is  a  significant  issue.  The 
primary  solution  being  explored  is  the  expanded  use  of  retrospective  pre/post  surveys  to  minimize  the 
number  of  survey  instances  and  shortening  the  survey  using  imputed  values  in  order  to  gain  the  necessary 
level  of  student  data  for  analysis.  Second  is  data  management.  Currently,  the  CSTEME  research  team  is 
working  on  transitioning  PIPES  data  into  a  relational  database  that  can  be  easily  queried  and  shared 
among  other  researchers.  Third,  the  PIPES  dataset  still  has  a  significant  self-selection  bias  on  the  data. 
Over  time,  CSTEME  personnel  have  been  able  to  minimize  this  bias  through  partnerships  with  school 
districts  and  pre-collegiate  development  organizations.  In  both  of  these  examples,  students  are  required 
to  attend  the  STEM  event  regardless  of  their  interest.  However,  many  PIPES  students  self-select  into  the 
program,  biasing  the  results  toward  the  positive,  creating  the  possibility  of  a  type-1  error.  The  CSTEME 
research  team  continues  to  explore  ways  of  achieving  quasi-experimental  conditions  related  to  STEM 
research  while  minimizing  confounding  variable  effects. 

In  summary,  because  of  the  AFOSR's  substantial  investment  in  the  PIPES  program  and  CSTEME,  there  now 
exists  a  robust  STEM  center  serving  Colorado  through  STEM  programs,  outreach,  research,  and 
evaluation.  Without  the  initial  investment  of  AFOSR  funds,  this  center  would  not  be  in  existence  nor 
would  it  have  achieved  the  level  of  sustainability  that  it  currently  enjoys.  When  the  PIPES  project  began, 
STEM  education  was  just  emerging  into  the  nation's  collective  consciousness  as  a  significant  need  for  the 
future  economic  viability  of  the  country.  Now,  at  the  close  of  the  project,  STEM  education  is  a  well- 
known  and  oft  talked  about  issue  in  the  US.  The  timing  of  the  AFOSR  funding  to  establish  PIPES  and 
CSTEME  was  superb  in  that  it  provided  a  means  to  examine  various  STEM  educational  strategies  while 
establishing  a  robust  longitudinal  database  that  will  serve  educational  researchers  for  years  to  come.  All 
of  these  outcomes  are  a  direct  result  of  the  funding  provided  by  AFOSR  through  the  PIPES  project.  With 
long-term  sustainability  in  place,  that  initial  investment  will  have  a  large  payoff  for  years  to  come. 
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APPENDIX  1  -  PIPES  WORKSHOPS 


Indoor  Flyers  -  Are  you  interested  in  airplanes  and  flight?  Join  this  workshop  that  explores  the 
mechanics  of  flight  by  building  your  own  ultra-light  rubber  band  powered  airplane  that  really 
flies. 

Solar  Car  Sprint  -  How  do  solar  panels  work?  Learn  how  by  constructing  a  working  solar 
powered  model  car  as  you  examine  photovoltaic  cells,  gear  ratios,  and  electronics. 

Bridge  Building  -  Are  you  a  bridge  builder?  Come  find  out  by  constructing  wood  bridges  to  see 
how  much  weight  your  design  can  hold. 

Green  Energy  -  You  are  hearing  about  "Green  Energy"  but  what  is  it?  Sign  up  for  this  workshop 
to  find  out!  You  will  construct  a  working  wind  generator  to  find  out  which  turbine  generates 
the  most  electricity. 

Computer  Game  Design  -  Do  you  like  computers  and  problem  solving?  Join  this  fun  workshop 
to  use  Robocode  software  to  develop  a  robot  battle  tank.  Battle  against  other  tank  creations 
in  a  contest  of  strategy  and  problem  solving. 

LEGO  Robots  -  Do  you  like  LEGOs?  Are  you  interested  in  robots?  Join  this  workshop  where 
you  can  combine  your  interests  to  build  and  program  a  LEGO  Mindstorm  robot  to  perform 
certain  tasks  on  a  playing  field. 

Advanced  LEGO  Robots  -  Do  you  have  some  experience  with  LEGO  Mindstorm  robots  and 
want  to  go  bigger  with  your  creations?  Join  this  workshop  to  build  and  program  an 
advanced  robot  that  will  accomplish  a  greater  number  of  amazing  tasks  using  additional 
sensors  and  commands.  Requires  some  previous  NXT  experience. 

Drawdio  -  How  do  music,  electronics,  and  creativity  relate?  Make  your  own  Drawdio  pencil  to 
find  out.  You  will  use  high  tech  tools  to  construct  a  circuit  that  turns  an  ordinary  pencil  into  a 
cool  electronic  musical  instrument.  Make  a  pencil  that  lets  you  draw  music. 

Animatronics  -  What  happens  when  high  tech  electronics  are  combined  with  your  artistic 
creativity?  Join  this  workshop  to  make  high  tech  "creatures"  that  respond  to  things  around  them. 
Rockets  -  What  does  it  take  to  launch  a  rocket  successfully?  Join  this  workshop  to  build  and 
launch  a  basic  Estes  rocket. 

Advanced  Rockets  -  Have  you  launched  rockets  using  small  A  engines?  Try  your  hand  at 
designing  and  flying  a  bigger  design  with  bigger  engines.  Requires  some  previous  rocketry 
experience. 

Card  Game  Design  -  Do  you  love  games?  Ever  thought  you  might  be  able  to  design  your  own? 
Join  this  workshop  to  explore  the  process  of  inventing  a  card  game. 

Cyber  Security  -  You  hear  about  "cyber  security,"  but  what  is  it?  Join  this  workshop  to  use  high 
tech  wireless  networks  and  cell  phones  to  solve  a  cyber-bullying  mystery  .  Learn  ways  to  stay 
safe  online. 
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Kitchen  Chemistry  -  Science  and  food,  what  could  be  better?  Join  this  workshop  to  explore  the 
role  that  chemistry  has  in  developing  artificial  flavors. 

Mousetrap  Vehicles-  Do  you  like  to  tinker  and  experiment?  Join  this  workshop  and  use 
common  household  materials  to  construct  a  race  car  powered  by  the  kinetic  energy  stored  in 
a  mouse  trap. 

GPS  and  Geocaching  -  You  hear  about  "GPS,"  but  what  is  it  and  how  is  it  used?  Join  this 
workshop  to  learn  how  to  use  GPS  technology  and  mathematics  to  find  hidden  treasure  on  a 
geocaching  course. 

Battery  Buggies  -  Calling  all  gear  heads!  Join  this  workshop  to  use  principles  of  mechanical 
engineering  and  electricity  to  construct  and  race  a  car  of  your  own  design. 

Wildlife  Biology  -  Do  you  love  being  outside  and  seeing  wildlife?  Join  this  workshop  to  explore 
the  roles  that  wild  animals  play  in  ecosystems,  and  engage  in  work  that  a  real  wildlife  biologist 
and  forester  will  engage  in. 

Veterinary  Medicine  -  Do  you  love  animals?  Does  a  career  working  with  them  sound 
interesting?  Join  us  in  veterinary  medicine  to  find  out  how  animals  are  treated  and  diagnosed. 

Crime  Scene  Investigation  -  Do  you  love  a  mystery?  Join  this  workshop  to  piece  together  a 
crime  from  forensic  clues  gathered  at  a  mock  crime  scene. 

Astronomy  -  What  is  the  closest  galaxy  to  ours?  Why  do  galaxies  hum?  Join  this  workshop  to 
learn  about  stars,  planets,  and  our  solar  system  using  high  tech  tools  of  astronomy. 

Physics  of  Filmmaking  -  Ever  want  to  break  through  glass  without  getting  hurt?  Turn  yourself 
into  a  superhero  by  making  a  short  movie  trailer  that  uses  digital  special  effects.  Explore  the 
physics  of  how  special  effects  work,  and  shoot  your  own  footage. 

Sports  Medicine  -  How  do  athletes  train,  and  how  does  a  physical  trainer  know  what  exercises 
an  athlete  should  do  to  be  at  the  top  of  his  or  her  game?  Examine  how  the  human  body 
responds  to  certain  activities  and  how  these  responses  are  measured. 

Animal  Behavior  -  How  do  animals  respond  to  their  environments?  If  you  are  interested  in  how 
animals  are  trained  and  behave,  join  us  for  this  interesting  workshop. 

Biotechnology  -  What  is  DNA  and  how  is  it  used?  Join  this  workshop  to  extract  and  see  your 
own  DNA  as  well  as  to  perform  other  experiments  related  to  biotechnology. 

Building  Trades/Civil  Engineering  -  Students  interact  with  the  science  and  math  behind 
buildings;  their  design,  and  construction. 

Health  Sciences  -  Thinking  about  medical  school?  Join  us  in  this  workshop  to  discuss  how 
doctors  diagnose  and  treat  diseases. 

Rocket  Chemistry  -  Students  interact  with  basic  chemical  reactions  including  how  glow  sticks 
work,  how  cold  packs  work,  and  other  common  chemical  reactions. 

Audio  Engineering  -  Are  you  an  aspiring  musician?  Join  us  in  this  workshop  to  examine  the 
physics  of  sound  in  musical  instruments,  and  apply  your  knowledge  to  your  own  digital 
recording. 
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Emergency  Response  -  Students  will  examine  the  science  and  technology  of  how  first 
responders,  like  police  and  fire  departments,  respond  to  emergencies. 

Math  Powered  Art  -  Students  will  create  their  own  unique  art  using  fractals,  Fibonacci  sequences, 
and  Pascal's  triangle. 

Cryptography  -  Do  secret  codes  interest  you?  How  does  computer  encryption  work?  Join  this 
workshop  to  learn  about  the  math,  science,  and  linguistics  behind  creating  and  breaking  codes 
and  ciphers. 

Liquid  Crystals  -  You  see  LCD  screens  everywhere,  but  how  do  they  really  work?  Join  this 
workshop  to  make  your  own  liquid  crystal  pixel,  and  understand  how  all  of  those  pixels  form  a 
high  definition  picture  on  an  LCD  screen. 

Chemistry  of  Smell  -  How  do  we  smell  things?  What  is  happening  in  our  brains  when  we  smell 
something?  Join  this  workshop  to  understand  the  chemistry  and  biology  of  smell,  and  become 
an  olfactory  engineer  by  designing  your  own  scents. 

Military  Science  -  Do  you  like  strategy  games?  Apply  your  strategic  thinking  skills  in  this 
workshop  to  outwit  your  opponents  in  a  friendly  battle  of  logistics  and  problem  solving. 

Snap  Circuits  -  Do  you  like  electronic  gadgets  and  wonder  what  all  those  little  parts  in  a  circuit 
actually  do?  Join  this  workshop  to  find  out  by  making  your  own  circuits,  and  find  out  what  all 
those  little  parts  actually  do! 

Arduino  -  Unleash  your  inner  inventor  by  learning  how  to  program  a  microcontroller  and  build 
circuits  for  robotics,  high  tech  art,  wearable  electronics,  and  all  sorts  of  smart  devices.  Learn  how 
to  program  in  C++  an  breadboard  your  own  circuits 
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APPENDIX  2  -  PARTICIPANT  COMMENTS 


Student  responses  to:  "My  favorite  part  of  this  camp  or  workshop  was:" 

•  "Trying  to  use  logical  reasoning  by  trying  to  solve  the  crime" 

•  "Being  able  to  visit  the  campus  and  use  their  tools" 

•  "Hands-on  projects  that  involved  current  issues" 

•  "I  love  deductive  reasoning  and  using  problem  solving  skills" 

•  "Military  science  was  the  most  hands  on.  I  believe.  Also,  everything  that  was  taught  could  be 
applied  to  real  life." 

•  "I  can  do  anything  if  I  put  my  mind  to  it" 

•  "My  favorite  part  of  the  Jumpstart  program  was  the  investigation  gathering,  evidence,  and  all 
of  the  different  lab  work" 

•  "I  love  the  UCCS  programs  in  general" 

•  "This  introduces  me  to  new  science  methods" 

Parent  comments  regarding  PIPES  Student  Programs: 

•  "My  daughter  enjoyed  the  program  so  much  that  she  continues  to  ask  me  when  the  next  one 
will  be.  She  is  set  on  attending  every  program  that  she  can.  I  am  excited  to  see  her  so  enthused 
about  learning!" 

•  "Great  experience,  she  enjoyed  the  class  (biotechnology).  Would  love  to  see  more  options 
(additional  dates)." 

•  "My  child  was  really  excited  when  he  came  home  from  PIPES.  He  really  liked  the  program  and 
was  amazed  by  the  things  they  learned  and  made  in  science.  He's  excited  about  attending 
more  PIPES  programs  in  the  future." 

•  "The  Sports  Training  Camp  opened  up  a  new  perspective  in  regards  to  science.  Excellent  for  my 
daughter." 

•  "[Participant  name]  had  such  a  great  time  at  STEM.  He's  asked  me  a  dozen  times  when  he  can 
participate  again... I  can't  thank  you  enough  for  putting  on  such  a  wonderful  program." 

•  "This  was  a  wonderful  experience  for  my  son.  He  had  not  felt  that  he  could  be  successful  in 
science.  He  especially  enjoyed  the  robotics." 

•  "The  most  interesting  was  the  field  trip  to  the  composting  farm.  He  told  us  all  about  the 
"farming"  and  the  mycelium.  He  was  so  interested  that  we  bought  a  composting  bin  so  he  could 
"grow"  rich 

soil.  He  continues  to  think  how  this  could  be  turned  into  a  community  service  project.  The  week-- 
and  especially  the  field  trip--was  inspiring  to  him  and  sparked  a  lot  of  interest." 

•  "[Participant  name]  enjoyed  learning  about  the  water  purifications  process.  And  her  music  pen 
is  still  working.  She  really  enjoys  it.  She  is  so  proud  of  it. 

•  "He  absolutely  loved  it!!!  The  hands-on  experience  is  everything!!" 
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Teacher  comments  regarding  PIPES  Educator  Programs: 

•  "Great! !  We  did  wonderful  activities." 

•  "The  sharing  of  both  their  [instructors]  educational  philosophies  and  techniques  were 
invaluable!" 

•  "OUTSTANDING!  I  was  extremely  intimidated  by  physics  before  this  experience;  Doug  did  a 
wonderful  job  helping  me  feel  comfortable  and  confident  in  learning  and  teaching  physics. 

•  "Very  beneficial  program" 

•  "THANK  YOU  for  the  opportunity  to  participate.  I  am  amazed  how  much  I  learned  in  this  past  2 
weeks.  The  team  was  very  dedicated  to  helping  us  to  be  successful  and  I  appreciate  their 
passion 

and  commitment." 

•  "It  was  a  great  professional  development  opportunity!  I  really  enjoyed  it." 

•  I  really  enjoyed  the  Math  Circle  and  felt  it  was  very  beneficial.  I  received  engaging  activities 
for  my  students  to  use  in  my  classroom.  I  liked  the  concept  of  sharing  ideas  with  other  Math 
teachers  and  having  lots  of  resources  to  help  with  Math.  I  feel  that  my  understanding  of 
Problem  Solving  has  grown  which  will  make  my  students  more  successful." 

•  "This  was  one  of  the  most  challenging,  rewarding  and  valuable  experiences  I've  had  in  my  math 
career.  The  time  in  the  mountains  made  me  feel  special  and  valued.  It  engaged  my  interest 
and  learning.  I'm  glad  we  had  a  chance  to  get  to  know  each  other  and  feel  "safe"  learning 
together.  I  feel  like  I  have  grown  and  have  become  so  much  better  equipped  to  teach  my 
students  to  persevere  in  problem  solving.  Thank  you!  Thank  you!" 
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APPENDIX  3  -  PROJECT  SURVEY  INSTRUMENTS 


Self  Efficacy  (SE):  Drawing  from  Bandura  (1977)  who  hypothesized  that  beliefs  about  self-  efficacy 
were  formulated  from  individual's  perceptions  from  four  sources:  mastery  experience,  vicarious 
experiences,  social  persuasions,  and  physiological  states  we  use  adapted  versions  of  the  Sources  of 
Mathematics  Self-Efficacy  Scale  (SMES)  originally  developed  by  Lent  et  al.  (  2001)  and  later  adopted  for 
use  with  science  by  Britner  &  Pajares  (2006). 

Sources  of  Math  Self-Efficacy  were  evaluated  by  adopting  the  24-item  Sources  of  Middle 
SchoolMathematics  Self-Efficacy  Scale  (Usher  &  Pajares,  2009)  for  use  in  this  study.  This 
scaleconsisted  of  4  subscales  listed  here  with  corresponding  Cronbach's  alpha  coefficients  reported  by 
Usher  &  Pajares:  mastery  experience  (.88);  vicarious  experiences  (.84);  social  persuasions  (.88);  and 
physiological  states  (.87).  Construct  validity  was  also  explored  by  Usher  &  Pajares  as  they  found  that 
the  items  both  individually  and  collectively  were  correlated  with  four  self-efficacy  measures  (i.e.,  math 
grade  SE,  math  skills  SE,  math  courses  SE,  &  self-regulatory  SE),  as  well  as,  self-concept  and  semester 
GPA.  Correlations  between  the  sources  and  SE  were  alL  statistically  significant  (p  <  .  001)  with  the 
highest  correlation  found  between  mastery  experience  and  SE  (.77). 

Sources  of  Science  Self-Efficacy  were  assessed  by  adopting  the  24-item  Sources  of  Middle  School 
Mathematics  Self-Efficacy  Scale  (Usher  &  Pajares,  2009)  for  science.  Britner  &  Pajares  (2006)  reported 
Cronbach's  alpha  reliability  indexes  as  .90  for  mastery,  .80  for  vicarious,  .88  for  social  persuasions,  and 
.91  for  physiological  state  when  adopting  the  scale  for  science  for  use  with  high  school  students  from 
the  original  scale  used  to  measure  this  construct  in  the  field  of  mathematics  in  college-aged  students. 

Math  /  Science  Self-Concept,  defined  by  Britner  and  Pajares  (2006)  as  "students'  perception  about 
their  science  ability  and  their  feelings  of  self-worth  associated  with  this  ability,"  were  evaluated  using 
a  6-item  science  scale.  Academic  Self  Description  Questionnaire  (ASDQ-1)  developed  by  Marsh  (1990). 
Alpha  coefficients  ranged  from  .88  to  .94  on  the  13  subscales  in  the  ASDQ-1,  including  the  science 
scale;  while,  Britner  &  Pajares  (2001)  reported  a  Cronbach's  alpha  coefficient  of  .82,  in  2006,  Britner  an 
Pajares  reported  a  coefficient  of  .89. 

Attitudes  toward  Science  and  Career  in  Science  were  measured  with  a  16-item  survey  designed  for 
use  in  the  current  study. 

Family  Encouragement  was  measured  with  a  4-item  scale  developed  by  Stake  &  Mares  (2001)which 
looked  at  student  perception  of  encouragement  from  family  members  for  science  pursuits.  Using  a  7- 
point  scale  1  (not  at  all  true)  to  7  (very  true),  internal  reliability  was  reportedly  .85.  We  have  elected 
to  use  a  6-point  scale  instead. 

Peer  Encouragement  was  assessed  with  a  7-item  scale  adapted  from  the  Friends'  AttitudesToward 
Science  subscale  developed  by  Simpson  and  Troost  (Owen  et  al.,  2008).  Through  factor  analysis  and 
cross-validation  methods  with  a  large  sample  of  middle  school  students,  Simpson  and  Troost  found  the 
subscale  distinctly  differentiated  from  14  other  school,  home,  and  self  variables  pertinent  to  science 
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evidencing  adequate  internal  reliability  (.71).  Stake  and  Mares  adopted  the  scale  using  5-items  for 
use  with  high  school  girls  and  found  internal  reliability  to  be. 70. 

Teacher  Encouragement,  also  adopted  from  Simpson  and  Troost  was  measured  with  a  6-item  scale 
adapted  from  the  Science  Teacher  subscale  using  the  same  procedures  described  for  the  Friends 
subscale.  The  original  scale  had  low  internal  reliability  (.44);  but,  Stake  and  Maresreported  adequate 
internal  reliability  (.79). 

Motivation  for  Science  Career  was  assessed  using  a  4-item  scale  developed  by  Stake  and  Mares(2001) 
to  evaluate  science  enrichment  programs  for  gifted  high  school  students.  They  used  a  seven  point 
scale  ranging  from  "not  at  all  true"  to  "very  true"  and  reported  an  internal  reliability  of  .93  at  pretest 
and  .95  at  posttest  and  follow-up.  We  used  a  six  point  scale  "not  at  all  true"  to  "definitely  true"  for 
the  present  study. 

Science  /  Math  Grade  SE  was  assessed  using  3-items  asking  students  to  provide  a  rating  of  their  level 
of  confidence  in  achieving  an  A,  B,  or  C  in  their  science/math  class. 

Math  /  Science  Aptitude,  defined  by  scores  on  standardized  tests  (i.e.,  CSAP/TCAP,  ACT)were  obtained 
from  student  transcripts. 

Math  /  Science  Achievement  was  defined  as  students'  grade  in  science/math  class  at  the  end  of  each 
grading  period. 

Science  and  Math  Career  SE  was  measured  by  asking  students  how  confident  they  are  that  they  will 
choose  a  career  in  each  field,  using  a  6-point  scale  ranging  from  "not  at  all  confident"  to  "completely 
confident." 

Course  Intentions  were  evaluated  by  asking  students  to  identify  STEM  courses  they  intended  to  take 
in  the  future  from  a  list  provided.  Other  researchers  measured  this  construct  similarly  (Lent  et  al., 
2001).  In  1993,  Lent  et  al.,  obtained  an  alpha  coefficient  of  .77  and  discovered  that  it  correlated 
significantly  with  math  self-efficacy,  interests,  and  math  ability.  Lent  et  al.  reported  an  alpha 
coefficient  of  .76  for  a  similar  course  intention  measure. 

STEM  Career  Interests  were  assessed  by  asking  students  to  list  3  jobs  they  might  like  to  have  when 
they  grow  up.  Students  were  also  asked  to  report  how  interested  they  were  in  19  different  STEM 
careers  using  a  6-point  scale  ranging  from  "not  at  all  interested"  to  extremely  interested."  This  method 
was  consistent  with  social  cognitive  research  on  math  outcomes  (i.e..  Lent  et  al.,  2001).  Lent  et  al. 
reported  an  alpha  coefficient  value  for  a  similar  measure  as  .84. 

Student  perceptions  of  program  impact  on  their  motivation  for  science  was  assessed  using  a  6-  item 
scale  developed  by  Stake  and  Mares  (2001).  Internal  reliability  was  .89  at  post  testing  and  .93  at 
follow-up. 

Student  perception  of  program  impact  on  their  science  confidence  was  assessed  using  a  6-item  scale 
developed  by  Stake  and  Mares  (2001).  Internal  reliability  was  .92  at  post  testing  and  .93  at  follow-up. 
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Student  perception  of  the  extent  to  which  the  program  increased  their  science  knowledge  was 

assessed  using  a  6-item  scale  developed  by  Stake  and  Mares  (2001).  Internal  reliability  at  post-testing 
was  .79. 

Student  perception  of  the  extent  to  which  the  program  helped  them  to  develop  a  network  of 
friendships  with  other  science  students  (a  new  social  niche)  was  assessed  using  a  5-item  scale 
developed  by  Stake  and  Mares  (2001).  Internal  consistency  at  post  testing  was  .85  and  at  follow-  up  it 
was  .83. 

The  Teaching  Science  as  Inquiry  (TSI)  was  validated  by  Smolleck,  Sembal-Saul,  and  Yoder  (2006);  and 
L.  A.  Smolleck  &  Yoder  (2008)  to  measure  teacher  self-efficacy  in  reformed  settings.  Overall  alpha 
coefficient  reliability  scores  in  relation  to  self-efficacy  reported  by  Smolleck  &  Yoder  (2008)  were  .94  at 
pre-test  and  .89  at  post-test.  In  relation  to  outcome  expectancy,  the  pre-test  alpha  coefficient  as  .90 
and  at  post-test,  .90. 

The  LSC  underwent  test-retest  analysis  by  Germuth,  Banilower,  &  Shimkus  (2003).  The  studyprovided 
correlation  coefficients  for  each  of  the  eight  composites  greater  than  0.60.  Based  on  these  findings, 
the  LSC  teacher  questionnaire  was  found  to  be  a  valid  and  reliable  measure  of  teachers'  attitudes, 
preparedness,  and  classroom  practices. 
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